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‘ABSTRACT. 


When analyzing evaluation strategies, one must systematically 
ements composing the various: approaches. © 


pick and choose among the el / 
While completing this task, it is important to keep in mind that 
ategy is to aid in evaluative de- 


the purpose of an evaluation str 
cision making. “Py utilizing a taxonomy approach, it is possible to 


analyze and compare various strategies in efforts to pick the ele- 
ments needed to satisfy a-given evaluative-need. ; 


/ 
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INTRODUCTION 

We all have our problems. . SO it is sfiedacartonaicomiuacors 

War; jJering the streets of academia in search of ap. Meret, educat iona] 
_ evaluators are caught in a conflict arising oe a sense of not belong- 
ing. | 

Whtther due to fear, frustration, or just a general lack of under- # 
standing, evaluation has never, until just recently, been given note than 
cursory acceptance. | 

Stufflebeam (1971) -reported, "Avoidance of evaluation is evidenced 
even within: the United States.Office of Education (USOE} which, histori- 
cally, has provided neither budget or staff sufficient to evaluate its 
own programs, “despite a great deal’ of talk about fis desirability of eval-. 
uation for the sciool."" [1] As in any “field where strict definitions are 
neither, appropriate nor possibte, evaluation begs the participant to con- 
stantly reassess his options in search of a better way. 

The problem arises when one tries to differentiate among. these better , 
ways. In-the field of evaluating progranis for casgdvantaged adults alone, 
the ERIC Clear inghousé on Adult Education lists 58 evaluation models cur- 
rently in use. 

By closely scrutinizing these as wéll as a sieehora- of otner strat- 
egies, one ‘finds a pattern otf similarities if not total repetition. In 
penne these various evaluative approaches, it becomes BPD arGINE Gee eval- > | 
uators ‘generally follow one of several basic patterns and then embellish 
the similarities with a personal touch in efforts to legitimize their ''new 

-ereatton. This approach ee in many respects, the expanding field 


of educat ional technology in gener and instructiehal development in 


particular. As in evaluation, instruétional developers have sled on a 


ERIC , 7e Se 4 


basic approach, focus ing on educational design, development, evaluition, 
and revisions, prior to ms ssaging additional steps geared to reflect their ; 
unique circumstances. - 7 | : : 
The same gencralization holds for evaluation--there is aie struc- 
ture‘that is .embeli ached to mect one's unique needs. The question arises: 
Is it possible to jepiate. particular, evaluative characteristics and then 
build, in cookbook fashion, a super-strategy based on the various major 
evaluative approaches? In essence, that is the purpose of this pdper--to 
analyze the major program: evaluation strategies utilized by the current 
leaders in, the field by developing a taxonomy of sais strategies. 
S - While delving’ into this analysis of evaluation strategies, heed a word 
of caution voiced by. W, James Popham, - a seasoned evaluation specialist 
from the University of California at Los Angeles ee "Although it is 
- sensible for educational evaluators to inform themselves of the nature of 
‘educational evaluation models, they should not get too caught up in that 
enticing. but enervating’ game known as comparative model meshing. Some 
‘people take great delight in Serine how Model X differs frem Model Q and 
is ever so eytentry like the seventy- -ninth stage ‘of Model Z. Instead of 
‘* 
engaging. in a game of ' ‘sames and’ differents, " she educaticnal evaluator 
sheculd become sufficiently conversant with the available models of eval- 
uation to decide which, if any, to mploy. Often a more eclectic - approach: =) 
“will t be adopted whereby one selectively draws fon several available models. 
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those procedures or constructs that appear most helpful." (2 
. - - a 


- 


2 Ot 
‘ 


WHY A. TAXONOMY? 


For the purposes of this paper, ‘'taxonomy' is definedts a system- 
per, , : 
/ : 


atic attempt’ at classifying phenomena into categories, either corre- 


r sponding to real ordering or to arbitrary ordering among the phenotien. 


Fl o« 
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Concerning the development of an evaluation design taxonomy, 


Worthen, (1968) states, "While manageable in cheery such-an appr one! Z. 


presents: the evaluator with a task which is extremely difficult. in 


practice. It requires that the selection of alternatives ees a 


balance between . selection of the most appropriate alternative. for each 


decision situation and ie deletion ofa set of alternatives, which. 


can be integrated into the dest over-all design." [3] 


In analyzing various evaluation designs, Stuifflebeam (1978) devel- 


oe 


oped : a logical evaluation rae structure that transcends all types of 


evaluation, whether content, input, process, or - product --the components 


fof his own aodel, [4] The ee elements are as follows: 


A. 


B. 


pos 


- 


<Focusthg ‘the Evaluation - so 

4. Identify the major level(s) of deci sior- -making to be served 
e.g., local, state, or national. : 

2. For each level of decision- -making , project the decision ‘situa- 
tions to be served and describe each. one. in terms of its locus, 
focus, timing, and composition of alternatives. 

Ss “Define criteria for each decision situation by. specifying vari- 

- abies for. measurement amd standards for” use in, the judgment. of” 

alternatives. « 

4. Define policies within which the evaluation must Ggcrate 


“Collection of Information a ot a 


_l. Specify the source of the information * 29 be collected. 


2. Specify the instruments and methods. for collecting the 
‘needed informaticn. « ‘ 

3. Specify the sampling procedure to be employed. , 

4, Specify the. conditions’ and schedule for * information collection. 


- Organization of Information ‘ 


1. Specify a format for the information which is. _to be collected. 
ce Specify a means tor coding, organiz ing, storing, .and retricving 
-infonnat ion. 


‘ 


“a. 


D. Analysis at inromntaon: ; c. a 


Lie Spee cify the analytical procedures. to he employed. : 
2x Specify a means for coding, organising, storing, and retricv- ee 
ing; information. 


. 2) 


E. Reporting. of Informition = . os ; * ’ “ s |. 


1. Define the audiences for the evaluation reports. . \ 

2. Specify means for providing information to the audiences. , \ 

a Specify the format for evaluition reports an Or; BepOrene: , 
*sessions. ~ . 

4. Schedule the feporting of satorintion. _ ; 


F. Administration of the Evaluation a 
1. Summarize the evaluation schedule = 
2. Define staff and resource requirements and plans for meeting © f 

* these requirements. — 


3. _ Specify means for meeting policy requirenents for conduct of the 
- evaluation. 


4, Evaluate the potential of the evaluation design for providing 


» °. information which is valid, reliable, credible, timely, and. . 
pervasive. 


Si Specify and schedule means for periodic updating of the evalua - 
tion design. 


6. Provide a budget for the total evaluation program. 


"These criteria along with others developed by the author: will be 


utilized in “development of this matrix-based evaluation taxonomy. 


“TEN EVALU EVALUATION STRATEGIES FOR COMPARISON 


, . For the purposes of chis taxonomy , the following evaluation strategies 


“were selected on the basis of eheae major ge aa and their relatively 
"unique approach to. this complete field. . The individual strategies along 
ro . with a brief annotation: of each are provided below:. ° 


- 


1. SAEs Comicon’ of Evaluation" ae 


wv 


Evaluation : data. are cither descr iptive--intents and Observations: , 
or judgmental. ‘In order to evaluate, an educator will oachien together 


‘certain data. Theadata are-likely ta be from several quite different 


en ey, 
woo 2 


sources, gathered in several quite different ways. Whether the immedi - 


ate purpose is description or judgment , three bodies of .informtion 


a . : aa 
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should be tapped. In the evaluation report it can be helpful to dis- 


tinguish anony antecedent, transaction, and catceme data. 


An antecedent is any condition existing prior to teaching and 
learning which may relate to outcomes. The status of a student prior 
to his lesson, c.g., his aptitude, previous experience, interest, and 


s , 
willingness, may comprise a complex antecedent. 
Transactions are the countless: ¢ encounters of aailewe: with teachers, 


students with student, author with reader, etc. Examples include the 
‘presentation-of a film, a class discussion, etc.. Transactions are 


. dynamic whereas antecedents and outcomes, are relatively static. 


‘Qutcomes are the a a of educat ing- - immediate ‘and long- . 
range, cognitive and conative, personal and “comiynity-wide. 
. : ’ u : i 
Judgnental statements’ are classified either as generitl. standards 


of quality or as judgments epee Pic to a given program. Descriptive 
ea data are classified as intents and’ Sheeriation®: The evaluitor. can or- 

, : , Be « ; : 

gins e his Gate gather ing to * conform to the format shown in Figure 1. {5} 
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ae a OUTCOMES ° it, 

a , DESCRIPTION MATRIX JUDGMENT MATRIX 

nr _« ° © (Figure 1) 
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a 
The Stake model, is considered to‘he the mest hwemmistie ef anv of 


the major models in current usc, placing heavy weight on judgsentil data. 
2. STUFFLEREAM; at DELTA KAP A\--"Decision- “Centered FE Evaluation (TPP) 


’ 


Evaluation provides informition for judging decision itt Oe: 
_It can be useful at all. stages. of decision miking--awareness,, design, 


choice, and action. Four levels of evaluation--content, input, process, 


and product --correspond to four major steps of programming decisions-- 


planning, structuring, implementing, and. recycling. 


#7 
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Oe. -PROVUS- ="Discrepancy ie }uintion"™ 


ley 


Evalyation jaspetties discrepancies between~ actual rprograms and 


st tandards for Progra so that programs can be improved. Program per- 


4 - bj 


sonnel set standards for’ activi ties aa results at each programni ing 


‘ ‘ S ‘ 


estace roc ayi installation, RIORESS and PEGE: “Actual perfonnines 


is compared with ‘de standaril § sand discrepancies of areas for LIproves 


sits yl 


ment identified. Discrepancy informaticn is need either to .chanze the 


+ 


‘ 


performince or to change the stindard. fe 


‘ 
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4. SCRIVIN:“"Goa! Free praluatont ey 


e . , 


Results Of programs are judged agaiiist the ‘originating neeu rather 


4 


ein the stated ghjcctives. Cost ‘of. producing those: results is consid- 


ered in relation to. costs at alternatives. The work” of evaluarion ‘is 


& 


viewed: as penal’ an act of condensat jon that ‘includes two major 


stages-- COMP EC SEIN and credentialing. A miss of data’ and observations 


~ 


are’ “compressed until: judgment’s as to the ae or worth of the program. 
can be made--credentiats, assi igned. [6] > 


x 
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S. SCRIVEN--"Formative and Summative Evaluation" 
There are two evaluative stages: formative--the purpose. of which is 

to assist in developing curricula--and summative--the purpose of which .is to 

assess the merit of curricula once they have been developed and are on 

tie market. Scriven states that formitive evaluation is an ongoing pro- 

cess ren constant feedback to the producer with summative evaluation 


being used to assess: the merit of the finished product by the consumer. 


6. GLASS--''Trade-off and Comparative Cost: Approach" 
A format for evaluation of materials or educational activities, .in- ~ 


cluding: describing the product to be evaluated, evaluating the goals of 


ae product; clarifying the point of entry of the evaluator; determining 


~ the kinds of. trade- offs involved; comparing costs with costs e ‘alterna- 


tives; making an intrinsic (secondary) evaluation; making an olitcome 
(primary) | evaluation; forming judgments and recommendations; “stipulating 
circumstances that would modify the conclusions; and evaluating the eval- 


uator. 


“7, HUNTER AND SCHOOLEY--"'The Synergistic Evaluation Strategy" 


An educational system can be broken down into four domains: (Tj: pol- 
icy, (2) program development, (5) instruction, and (4) feedback. Each 
domain contains activities which are of a technical and non-technical 


nature which comprise a complete evaluation strategy. The approach is 


synergistic in three ways: (1) its activities require the involvement and 


complete interaction among students, educators, and parents; (2) it requires 


the interactign betweer'technical and non-technical aspects of evaluation; 


o_ 
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and (3) its activities incline both goal based evaluation (GbE) and goal 
. 5 a > é é 


~ free evaluation (GFE). [7] | 


8. FOSTER--"The Karlsruhe Evaluation Strategy" 
As an offshoot of the Scriven Formative/Sumative evaluation system, 
the basic steps involved are: (1) preplanning, (2) planning and model 


eeverepnent. (3) implementation, (4) operation, anc (S) dissemination. 


In contrast to most. developmental strategies, ¢ evaluation is not listed 


as the final phase because evaluation is viewed as an ongoing activity. 
at all stages of program development. 


sos 


9. CREMEL--"'The rene) Evaluation Strategy" 


The Cremel strategy is developed utilizing an X/Y matrix. The X. 


axis is,concerned with the following cutegories: ” (1} issues, (2) initia- 


Pe 
(3) pilot, (4) field test, and (5) public diffusion. The Y axis 


-is concerned with: (1)° desirability/feasibility, (2) management pro-- 


eeitial cost, (5) product worth, (4) usauLtynys and (5) generaliz- 

ability. The evaluator responds to the matrix in terms of: (4) eri 
teria, and (2) audicnce. This enables those involved in the study to 
analyze the individual segments of the evaluation as to their instruc: 


tional: worth... 


x 


v4 
ee 


In: using’ the Creme]: strategy, the*evaluator helps eie’eiene ta ’ 
identify the issucs .to be considered, i.e., "Desirability/¥easibility." 
The evaluator/client’ team then identities the various audiences for each 
issue (varies from issuc to issuc) and the criteria needed to satisfy 
a’ particular audience in regards to a specific teaue, Similar criteria/ 


audience a is eonduchet for the \ various Stages of program evalua- 


tion, ee initiation, pie: ficld testing, public diffusion. (Figure 2) 
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‘ : SEQ] NT OF CEIMEE STAITGY? ; ae . 10 
. (fig. 2 ) ae + . 
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yeay ort 


ve Wag are the 


Supt feeds 
of the uni- 


versitve 


» Is this prot: 
len a high 
mriority 
issue? 


y Are there 
products.” 
avi ilable 
“or-in de- 
velopment 
which might 
Abe adapted 
to this 
purpose? 


1. What type of 
product is : 
neccssary, £0 . 
fill the: ar 
~ need? 
eo. Identifi-* 
1 cation of 

a 


target pop- 
ulation, 


: 10. STUPFLEBEAM/SCRIVER/GUBAs -""Mota “Eva aluation, or, the “Fi aluation of 
Evaluations" : 


Because of its varied anEGKERES ations, a little background infor- 
mition on Meta- Evaluation is in order. The term 'Meta-Fvaluation" 
means "A procedure for describing an evaluation activity and compar- 
ion it against a set of ideas concerning what CONSE MUA SS good evalua 
tion,’ According to Scriven, this incans that mota- evaluation is higher 
order than standard evaluation and includes evaluations that are dae: 

ondary, tertiary, etc. [8] ; 


Criteria for judging meta - evaluation have been discussed in the 


works et Campbell and Stanley [9]; Gephart,, Ingle and Reinstad 1 [10] 


and. Sak ‘and Giiss. mi 
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A MetgeEvrintstia Fricawork 
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The sti -ucturing of meta- “evaluation appears in Figure 3. This 


structure portrays meta-evaluation as a. methodology for assessing the 


merit of proposed aa complet ed evaluation efforts as discussed in the 


H 


first premise. | The framework relates to three diftens ions, namely; 


PURPOSES, OBJECIS, AND STEPS: .. The contents of the cells of a struc 


ture reflect ape criteria of eamien-areanas utility and cost/ 


ee ee ‘Meta- evaluation follows the premise that insiders should 


conduct formative or pro active meta evaluation while external ‘agents 


: 
“ 


~ should conduct summative ; ‘or retroactive metas ~eva lit ion. . POA 2 


" seufflebean sti ites that there are two > specific purposes of meta- 


ie 


ey 


aa 


evaluat ion: 12] . -. 2 i 


_ “ ae 
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1. Meta-evaluation should serve decision miking and accountabil - 


itye In order to support dec ision nS in evaluation, it Rey 


ding emphiasi = cs, the need for formative ork din efforts to provide ante. 


. , 


recommendations concerning how evaluation. etd ies should be designed 


and carried Out. Formative meta-eviluation is ‘a direct way of insuring 
that evaluations. will produce meaules that are ‘technicall¥ adequate , 
useful, and cost/cffective. 


2. The second, and possibly outwardly more important, purpose of 


meta-evaluation is to provide a system of acceuntability for the cval-. 


* 


uator. To facilitate this accountability, meta-evaluation of a sum- 


mative type is conducted ey outside agent, to produce an un- -biased out - 


. side opinion and foster public suspnents of the merits “of the eommncee 


evaluation work. Much of the information required in sunnat ive meta - 

, y 
evaluation is potentially available in the formitive meta-cvaluation. 
; rete ae 


"MATRIX FORMAT as : 


An assessing “the various OV -aluation strategics, several character- 


_ 


istics were deemed necessary to explore. They are: 


« 
a 


A. Type of Evaluation--The context in which the evaluation is conducted. 


B. Definition--The pr ieny aim of the evaluation. 


C. “Purpose -thirt the results of the evaluation will be used for.’ 


D. Criteria for Judging Evaluttion--The criteria by which the evelun- 


wa 


tive design cand alae eed will be “judged. 
E.' Personnel] Required--Staffing implications in rejJatian to a given 


“evaluative pematesy i.CQ.,5 to rice Cube sata criteria 


& 
certain strategies necd more manpower than others. (This is an inm- 
Ne 
porisit budgetary: consideration.) 


= - - 14. 
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vious, others remain, hidden for the most part. i # See 


_ more conprehensive, they would ‘aiso tend ‘to make the taxonomy more difti~ 


Bes nt Tea von for Design--Specific design constraints encountered when 


following a given strategy. 


G. kev Emphas is- -How the information gleaned. from the evaluation is used. 


Hl. Role of Evi luaton--The- evaluator’ s role in regards to a knecific 


strategy. 
I. Relationship to Objectives--Mcans by which the objectiva sare amet. 


a3 Relationship to Decision Making- -How the eval utEton strategy scrvés.. 


decision making. 


nn 


- Time to Cofplete--Approximate time needed to complete a. given aon 
uation. Time varies greatly concerning one size of the program an 
_ depth of the: evaluation. _ ph ’ 
Le Cost--Budgetary requirements in very genial terms dependent: once 
again, on the size ‘of the orepean and depth of: the evaluation. 
It is quite apviods that various: other characteristics eourd be 4: 
plugged ‘into this framework. It is felt, however, that the charger 
istics listed here will- provads: the reader with a firm base trom which ‘ 


various evaluation strategies can be analyzed. 


LIMITING FACTORS , | a Oo _— 


Every taxonomy has inherent drawbacks. While-some are quite ob- e 


In looking over the characteristics considered i in the « matrix, it is 


apparent that ‘other aspects of the models could have easily been added | 


to this “framework. While additional characteristi¢s would | make the. taxonomy | 


Poa 
——— 


cult to digest. Another problein arises from the difficulty of classifying 


various, strategies using Somen ee vague , terminology such as "cost, _ 


_ age t a is. 


ad 
4 ° , 5 : ye - - 4 ° 
"purpose,'' ete. Terminology that is too rigid will tend to misrepresent 
certain strategies while non-specific terms will fail to.give a firm base 
; o 
by 8 which different Strategies C can be analyzed. 
A final factor of concern to the person actually conduct ing an eval - 
: uation is the size of ne program be ing analyzed. . one should keep in 
s z : . ; ; : i . x8 
mind. that the size of an evaluative effort, -both in terms of budget, per-- 
‘sonnel requirements, deadlines, etc., will have a great affect on the a 
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Almost. all. mode 1s include the format ive/ summative evaluative process | 
evedited to Scriven. Although Scriven ee ‘ed the. term "form ative/ 


summative -evaluation;' " the vast majority of present day’ aviluative strat-. 


< paies beter that time utilize ed some, form of program feedback for guidance 
from project initiation. to comer ne 
Other approaches, such as ‘Stake’ s "Countenance of bytnuation system 


attempt to “gather ae data utilis ing different nosloass such as Stake' 


humanistic approaéh to eathering information. ie 
An example of evaluating a given product er program using a unigue 
- approach is Scriven's "Goal-Free'! evaluation strategy in which goals, the 


guiding light of most evaluators, are intentionally. by-passed in efforts. 
“to create an outcome-based, bias-free evaluation. - 3 ; _ _ 
: Early ‘in the deve lopment of this tuxonomy, the need was expressed to 
find a "super strategy" utilizing the best elements of each evaluative an- 
' . _s . 


proach. In. analy: ing the approaches: discussed “in this paper » it becomes 


. hy 


obvious that the developnesit ‘of a “super strategy" for educational eval - 


uation is not a new concept... 
: L — a, 


. . . Each of the strategies discussed in this paper is an attemt at devel- 


Fa 


oping the ultimate evaluation tool de ered to fill a specific need. The 


= a ‘ 


re 


ec 
=< 5 


Techno logy (1963). - 


‘ . : : " . i 
success of various strategists in reaching this end can only be decided, 


= . 


by the individual evaluator, who, after assessing his evaluation needs, 


must. look at evaluation options currently available and decide whether to 


& 


. : 
utilize an existing one or develop a “sumer strategy" of his own which 
transcends previous nodels. 2 

: : 


Qne needs’ to ene however, that evaluation strategies need not 


be complex to be effective.» There 1s, for example, 2 a particularly ‘anes 


: : 
one devéloped by* Ernest R. House, and reported in-his article: "Confessions 


of a Res sponsivé Goal- Free Evaluator.'" Cloaked in the jargon of educat iona t 


evaluation, the eenmiesin- cheek House states, "Below is an evaluation mode 1 


developed by an- Office’ of Education official and ‘reported in Educational 


‘5 


Or was it an eva aluation ‘model of the ‘Orfice of Education? . In any case, 


the data will be atrayed into the above ‘scheme."' [18] 
‘House’ s point is well taken. The purpose ot an cducational ovd Lua: 


tion strategy should be to facilitate evaluative decision making. This 


a . 


should be done as“ efficiently. as possible without undue glorification and: 


obfuscating frills. While House's simplistic model overstates the case, 


the measure of a " good" evaluation strategy is its ability to fi11 an 


eaprCseee evaluative need and not to add More jargon to a field approaching 


. 


the saturation point. 
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